Verifying the Solution from a Nonlinear Solver: A Case Study: Reply

We are pleased to confirm that any doubt our article (McCullough and Vinod, 2003; hereafter “MV03”) may have cast on Ron Shachar and Barry Nalebuff (1999; hereafter “SN99”) must be removed. We are especially pleased because we thought it quite unfair that other researchers were able to exempt themselves from such detailed scrutiny. It appears that such researchers no longer will have the luxury of reneging on their agreement to honor the replication policy, as this journal now requires authors of accepted empirical papers to provide all programs and data files for posting on the AER Web site as a precondition of publication. The primary aim of our article (MV03) was to provide a four-part methodology for verifying the solution from a nonlinear solver: check the gradient, examine the trace, analyze the Hessian, and profile the likelihood. We adduced copious evidence (MV03, p. 873) that solvers used by economists can produce inaccurate answers, gave examples of different packages giving different answers to the same nonlinear problems (MV03, p. 874), and showed (MV03, pp. 873–74), at least in this journal, that researchers make no effort to verify the solutions from the solvers that they use. We believe this uncritical acceptance of solutions from nonlinear solvers to be a systemic problem in economic research; that is why we wrote the article—certainly, econometrics texts do not show how to verify the solution from a nonlinear solver. In passing, we also showed how a problem can be too large for conventional PC methods, and indicated the failure of replication policies in this journal and other journals. We used the data and likelihood function from SN99 to illustrate the methodology. In the course of this illustration, we noted that the Hessian was ill-conditioned, suggested that there might exist multiple optima, and that inference based on the Wald statistic was not appropriate. We concluded that the solution we found was, at best, a tentative solution. However, as shown in Shachar and Nalebuff (2004; hereafter SN04), when the problem is rescaled the Hessian is not ill-conditioned; they have correctly identified the difference between the condition number of the badly scaled version of the problem that we analyzed in MV03 and the well-scaled problem that they have analyzed. When the problem is correctly scaled, the Hessian is well-conditioned, the model is locally identifiable, the problem can be solved on a PC, a solution to the problem exists, and SN present it in their Table 1. Though we were aware that rescaling could ameliorate ill-conditioning (MV03, p. 882), we were unaware of the distinction between artificial ill-conditioning and inherent ill-conditioning, so the method we suggested for analyzing the Hessian contained an error of omission. This error caused us to reach an incorrect conclusion concerning the existence of a solution to the problem. We apologize to Professors Shachar and Nalebuff, and we thank them for their gracious understanding in this regard. Accordingly, we have amended our prescription for analyzing the Hessian—see our nearby exchange with David M. Drukker and Vince Wiggins (2004; hereafter “DW”) for complete details. Our